Papers with neural language model
Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Coding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Linguistic steganography studies how to hide secret messages in natural language cover texts. |
| Approach: | They propose a method which encodes secret messages using self-adjusting arithmetic coding based on a neural language model. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on four datasets by 15.3% and 38.9% in terms of bits/word and KL metrics. |
Is Partial Linguistic Information Sufficient for Discourse Connective Disambiguation? A Case Study of Concession (2025.acl-srw)
Copied to clipboard
| Challenge: | Discourse relations are often not linguistically marked, but there are various connectives that explicitly signal discourse relations. |
| Approach: | They analyze linguistic features that play an important role in disambiguation of polysemous connectives in Japanese by performing a neural language model. |
| Outcome: | The proposed model performed well after removal of one of the two arguments that constitute the discourse relation, but significantly degraded disambiguation performance. |
Sensei: Self-Supervised Sensor Name Segmentation (2021.findings-acl)
Copied to clipboard
| Challenge: | Sensor names are alphanumeric strings that encode key contextual information such as their function or physical location. |
| Approach: | They propose a self-supervised framework that can learn to segment sensor names without human annotation. |
| Outcome: | The proposed framework can learn to segment sensor names without human annotation on buildings. |
How to Avoid Sentences Spelling Boring? Towards a Neural Approach to Unsupervised Metaphor Generation (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to generate metaphors rely on template-based or rule-based knowledge, which constrains the diversity of generated metaphors. |
| Approach: | They propose a neural approach to metaphor generation that uses wiki corpus to extract metaphorically used verbs and train a language model. |
| Outcome: | The proposed approach generates metaphors with good readability and creativity using wiki corpus and automatic metrics and human evaluations. |
Decipherment of Substitution Ciphers with Neural Language Models (D18-1)
Copied to clipboard
| Challenge: | Existing methods for deciphering homophonic substitution ciphers use pre-trained neural LMs. |
| Approach: | They propose a beam search algorithm that scores the entire candidate plaintext at each step of the decipherment using a neural language model. |
| Outcome: | The proposed beam search algorithm improves on challenging ciphers with smaller beam sizes and better error rates than state-of-the-art methods. |
Evaluating pragmatic abilities of image captioners on A3DS (2023.acl-short)
Copied to clipboard
| Challenge: | Evaluating grounded neural language models with respect to pragmatic qualities such as truthfulness, contrastivity and overinformativity remains a challenge in absence of data collected from humans. |
| Approach: | They propose to use an open source image-text dataset to evaluate pragmatic abilities of grounded neural language models with respect to pragmatic qualities such as truthfulness, contrastivity and over-informativity. |
| Outcome: | The proposed model develops human-like pragmatic abilities with respect to truthfulness, contrastivity and over-informativity for specific features. |
Multi-Word Lexical Simplification (2020.coling-main)
Copied to clipboard
| Challenge: | In text simplification, individual words are replaced with their simpler equivalents, but single word substitutions do not cover the full complexity of techniques humans use to approach text simulating. |
| Approach: | They propose a task of multi-word lexical simplification in which a sentence is made easier to understand by replacing its fragment with a simpler alternative. |
| Outcome: | The proposed method is based on a purpose-trained neural language model and evaluates against human and resource-based baselines. |
Unsupervised Paraphrasability Prediction for Compound Nominalizations (2022.naacl-main)
Copied to clipboard
| Challenge: | Nominalizations can be difficult to interpret because of ambiguous semantic relations between deverbal noun and its arguments. |
| Approach: | They propose to over-generate clausal paraphrases to predict whether a prenominal modifier can be re-written as a noun or adverb in a claual paraphrasability. |
| Outcome: | The proposed method improves paraphrasability prediction and paraphrase generation in English . it shows that the prenominal modifier can be re-written as a noun or adverb in a clausal paraphrase . |
Truncation Sampling as Language Model Desmoothing (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Long samples of text from neural language models can be of poor quality. |
| Approach: | They propose to think of a neural language model as a mixture of k and a true distribution that avoids infinite perplexity. |
| Outcome: | The proposed methods generate more plausible long documents according to humans and break out of repetition. |
Hidden Schema Networks (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models that encode rich semantic and syntactic content are biased, but they are effective at encoding symbolic representations. |
| Approach: | They propose a neural language model that enforces explicit relational structures which allow for compositionality onto the output representations of pretrained language models. |
| Outcome: | The proposed model can encode sentences into sequences of symbols and infer the posterior distribution of the model from natural language datasets. |
Towards Zero-shot Language Modeling (D19-1)
Copied to clipboard
| Challenge: | a number of natural questions have been asked about the inductive biases of neural networks on core NLP tasks. |
| Approach: | They construct an informative prior for held-out languages on a task of character-level, open-vocabulary language modelling. |
| Outcome: | The proposed model outperforms baseline models with an uninformative prior in both zero-shot and few-shot settings, showing that it is imbued with universal linguistic knowledge. |
Discourse-Aware Soft Prompting for Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in pre-trained langauge models (PLMs) have made great impact on text generation research. |
| Approach: | They propose to use hierarchical blocking to simulate a higher-level discourse structure of human written text and attention sparsity to learn sparse transformations on the softmax-function. |
| Outcome: | The proposed methods perform better on some generation tasks but don't generalize across all generation tasks. |
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered . |
| Approach: | They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets . |
| Outcome: | The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark. |
The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding (2023.findings-acl)
Copied to clipboard
Hao Fang, Anusha Balakrishnan, Harsh Jhamtani, John Bufe, Jean Crawford, Jayant Krishnamurthy, Adam Pauls, Jason Eisner, Jacob Andreas, Dan Klein
| Challenge: | In a task-oriented dialogue system, response generation is a conditional language model, but effective dialogue agents must balance fluent generation with stricter constraints. |
| Approach: | They propose a rule-based content selection model that transduces a dialogue agent’s actions and their results into context-free grammars representing the space of contextually acceptable responses. |
| Outcome: | The proposed architecture outperforms both rule-based and learned approaches in human evaluations of fluency, relevance, and truthfulness. |
PaLM: A Hybrid Parser and Language Model (D19-1)
Copied to clipboard
| Challenge: | Recent language models have shown strong data-fitting performance, but do not explicitly encode any notion of structural information. |
| Approach: | They propose a hybrid parser and neural language model that adds an attention layer over text spans in the left context. |
| Outcome: | The proposed model outperforms baseline models on language modeling and provides syntactically-informed representations of the context. |
Solving Historical Dictionary Codes with a Neural Language Model (2020.emnlp-main)
Copied to clipboard
| Challenge: | a dictionary-based substitution code is common, but no automatic decipherment algorithms exist. |
| Approach: | They propose a decoding lattice and a neural language model to solve word-based substitution codes . they apply their method to letters exchanged between general James Wilkinson and agents of the Spanish Crown . |
| Outcome: | The proposed method decrypts letters written by general James Wilkinson and agents of the Spanish Crown in the late 1700s and early 1800s using a neural language model. |
A Neural Model of Adaptation in Reading (D18-1)
Copied to clipboard
| Challenge: | Several studies suggest that readers do adapt their lexical and syntactic predictions to the current context. |
| Approach: | They propose to add a simple adaptation mechanism to a neural language model to improve predictions of reading times. |
| Outcome: | The proposed model improves predictions of human reading times compared to a non-adaptive model. |
Automatic Nominalization of Clauses through Textual Entailment (2022.coling-1)
Copied to clipboard
| Challenge: | Past research on clause nominalization has focused on replacement of the head verb with a deverbal noun and resource development to support the task. |
| Approach: | They propose to use a textual entailment model to optimize the position and POS of nominal arguments by fine-tuning a model on the task. |
| Outcome: | The proposed model outperforms unsupervised approaches on the nominalization task and outperformed a state-of-the-art neural language model. |
Can Sequence-to-Sequence Models Crack Substitution Ciphers? (2021.acl-long)
Copied to clipboard
| Challenge: | Current methods for deciphering historical ciphers use beam search and a neural language model . but, this approach assumes that the target plaintext language is known . |
| Approach: | They propose an end-to-end multilingual decipherment model that can solve 1:1 substitution ciphers without explicit language identification. |
| Outcome: | The proposed model can decipher text without explicit language identification while still being robust to noise. |
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (P19-1)
Copied to clipboard
| Challenge: | Existing language models are only capable of remembering facts seen at training time, and have difficulty recalling them. |
| Approach: | They introduce a knowledge graph language model with mechanisms for selecting and copying facts from a Knowledge graph that are relevant to the context. |
| Outcome: | The proposed model outperforms a baseline language model in generating factual knowledge and generating sentences that require factual information. |
Neural Language Modeling for Named Entity Recognition (2020.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that named entity recognition systems are faster and more flexible for the size of the corpus. |
| Approach: | They propose to use a neural language model as an alternative to the conditional random field layer for named entity recognition. |
| Outcome: | The proposed system has a significant speed advantage with a marginal performance degradation. |
Connecting degree and polarity: An artificial language learning study (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that degree modifiers are related to sentence polarity, but they are not related to the grammatical number of an expression. |
| Approach: | They propose to generalize degree modifiers to their polarity sensitivity in pre-trained language models by applying the Artificial Language Learning experimental paradigm from psycholinguistics to a neural language model. |
| Outcome: | The proposed generalisations are consistent with existing linguistic observations that relate de-gree semantics to polarity sensitivity, including the main one: low degree semantics is associated with preference towards positive polarities. |